Visual Features Involving Motion Seen from Airport Control Towers 
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Abstract: Visual motion cues are used by tower controllers to support both visual and anticipated 

separation. Some of these cues are tabulated as part of the overall set of visual features used in towers to 
separate aircraft. An initial analyses of one motion cue, landing deceleration, is provided as a basis for 
evaluating how controllers detect and use it for spacing aircraft on or near the surface. Understanding cues 
like it will help determine if they can be safely used in a remote/virtual tower in which their presentation 
may be visually degraded. 
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1. INTRODUCTION 

The visual cues necessary to fly and land an aircraft have 
been well studied over many decades (e.g. Gibson et al , 
1955; Grunwald & Kohn, 1994). In particular, the degrada- 
tion in piloting performance and the consequent need to re- 
duce airport capacity due to bad weather is fairly well under- 
stood. (FAA 71010.65R, 2006). The present report outlines 
an approach to a complementary side of the airport capacity- 
safety trade-off quantifying the visual features and properties 
used by tower controllers. These features are now 
particularly interesting due to recent proposals for 
technology and procedures in which controllers lose visual 
contact with their controlled space as in a “virtual tower” 
physically removed from the airport (JPDO, 2007). 

What important visual information would be lost in a virtual 
or remote tower and how may it be replaced by new air- 
craft/airport sensors or managed by new procedures or dis- 
plays? Could controllers return to some form of visual pro- 
cedures were these new sensors and associated displays and 
procedures to fail? These are some of the questions ulti- 
mately needing answers. 

The following discussion of these questions will first point 
out relevant visual elements of the control task facing the 
tower evident in previous task analyses (see below). How- 
ever, this earlier work appears to only provide very general 
descriptions of the specific visual features to which that the 
controllers attend. To the extent the visual functions import- 
ant to the controllers are discussed with precision, they are 
generally limited to questions of detection, recognition and 
identification. The following discussion will consider other 
visual features, in particular motion of the controlled aircraft. 
The preliminary conclusion of the discussion is that tower 
controllers use visual features to provide predictive position 
information allowing them to use anticipated separation to 
effectively and safely merge and space aircraft to maximize 
airport capacity. 


The visual cues used by controllers are important for several 
reasons. In the first place, there is FAA interest in increasing 
airport capacity so that current operations under non-visual 
flight rules with reduced capacity may be modified to allow 
higher capacity visual operations. For this purpose the cur- 
rently used visual information needs to be provided by alter- 
native means. Such “Equivalent Visual Operations” described 
FAA/NASA planning documents may be achieved with syn- 
thetic visual system, i.e., (Kramer, Williams, Wilz & Arthur, 
2008) visual replacement with visualized electronic position 
data. But these replacements will not be fully successful, and 
may even be tragically misleading, if the useful visual 
affordances provided by the real scene are not appropriately 
provided. Although Equivalent Visual Operations has 
primarily been thought of from the pilot’s viewpoint and 
discussed in terms of flight displays using new sensor data for 
synthetic vision, it has a flip-side for which synthetic vision 
or camera-based displays could present useful visual features 
in a remote or virtual tower. 

Significantly, this information need not be provided in the 
form of an image, but could be provided in a more map-like 
plan view format and conceivably could even come along non- 
visual sensory channels, e.g. auditory or haptic. In fact, it 
could be based on data directly down-linked to ground dis- 
plays from the aircraft indicating its state, e.g., spoilers de- 
ployed (Hannon, et al., 2008). 

The visual environment in an airport tower may be illustrated 
by considering the view from a tower such as that of San 
Francisco International Airport (SFO). Such tower views 
show significant perspective compression at the ~1 n.mi. 
range to runways and taxiways, making commercial aircraft 
subtend small visual angles, and posing viewing difficulties 
due to background visual clutter. Interestingly, during low 
visibility CAT III operations at SFO, airport operations may 
be conducted with the controllers never actually seeing the 
aircraft. Thus, since it is already possible for the controllers 



to continue many of their control tasks without visual con- 
tact, the idea of a remote tower may have merit. But without 
visual contact, they must inform the pilot and those monitor- 
ing their communications that visual contact has been lost. 
At the SFO tower continued operation without visual contact 
is associated with a significant loss (-50%) of airport ca- 
pacity. In contrast at an airport such as Arlanda, Sweden 
(ARN) with the parallel runways ~1 km apart, total loss of 
visual contact can have virtually no impact on capacity when 
ground radar is properly functioning. 

2. SFO OPERATIONS 

A more specific analysis of the role of visual features in 
tower control can be developed from a more concrete discus- 
sion of operations for a particular airport, SFO. A sense of 
the overall strategy for some aspects of usual airport oper- 
ation at SFO is best gotten from plan- views maps (Figures 1 
and 2). Aircraft are taxied from their gates to the south ends 
of runways 1L and 1R and launched in staggered pairs that 
are interleaved between approximately paired aircraft land- 
ing on Runways (RW) 28L and 28R which taxi to their gates 
crossing to the southwest. (Figure 1.). Current winds, 
weather, and special operational requirements, of course, can 
significantly alter this pattern. Sometimes the longer 28 
runways are needed, for example, for departing heavy trans- 
pacific aircraft. Detailed descriptions of the approach and 
departure procedures can be found in the Standard Instru- 
ment Departures (SID) and Standard Terminal Arrival 
Routes (STARS) associated with the airport, but the local 
controller’s responsibility generally begins with radio con- 
tact somewhat before the aircraft crosses the San Mateo 
Bridge and ends 1 n.mi. beyond the departure runway. By 
FAA rules, the local controller is generally responsible for 
aircraft entering and leaving the runways whereas the ground 
controllers handle most of the taxing to and from the gate. 
These two positions in addition to that of the supervisor are 
the ones that make the most use of the out-the-window in- 
formation. The Flight Data and Clearance Delivery con- 
trollers primarily use inside-the-tower information sources 
and voice communications. 

3. VISUAL INFORMATION IN THE TOWER 

The primary responsibility of the control tower is to en- 
sure sufficient runway separation between landing and 
departing aircraft. (FAA, 2006) A back propagating pro- 
cess may be used to understand the visual requirements sup- 
porting the tower controller’s primary responsibility. 

This process first identifies the visual affordances that the 
controllers’ tasks involve. Affordances are the higher-level 
behavioural capacities that vision must support. Controllers, 
for example, must identify the aircraft type, company and 
flight status. They must control and recognize aircraft 
speed, direction and position. They must establish a move- 
ment plan involving a succession of spatial goals. They 
must communicate this plan to the aircraft, coordinate it with 
other controllers and pilots as necessary, establish whether 
aircraft comply appropriately, and recognize and resolve 
spatial and other conflicts that may arise. These higher-level 
elements are supported visually by a number of visual func- 


tions: detection, recognition, and perception of the static and 
dynamic state of the aircraft. These functions are supported 
by still lower level visual mechanisms underlie luminance, 
color, control, position, and movement pro-cessing. These 
three levels of analysis provide a basis for describing the 
controllers visual task. 



Fig. 1 SFO airport diagram 
showing typical movement 
paths for United Airlines, de- 
partures (dark/red), arrivals 
(light/ green) 


Fig. 2. The first and last posi- 
tions where SFO controllers 
report useful visual informa- 
tion w/r to landing (RW 28) 
and departing aircraft RW 1). 


The tower controller’s overall task has, of course, been ana- 
lyzed within and outside of the FAA. It may be broken down 
to six different job sub tasks: separation, coordination, control 
judgment, methods/procedures, equipment, and communica- 
tion. The five of these subtasks which involving vision have 
been identified by boldface type in Table 1. 


The assurance and maintenance of spatial separation is, of 
course, a visual task since regardless whether separation is 
determined by radar or direct view, it is definitely recognized 
visually. Handoffs and point-outs clearly are also intrinsi- 
cally dependent upon vision, though the need for the control- 
ler to adopt the pilot’s spatial frame of reference to direct 
attention toward objects and aircraft is also a significant cog- 
nitive task. Control judgment, being essentially a mental and 
cognitive issue, does not have an intrinsically visual compo- 
nent. But its connection with maintenance of effective and 
efficient traffic flow does emphasize the critical importance 
of time in traffic control. Three general methods and proced- 
ures directly involve vision. These include establishment and 
maintenance of aircraft identify, posting and correct annota- 
tion of flight strips, and continual scanning of the entire con- 
trol environment. Associated with these methods is the ad- 
monition to work quickly and rapidly recover from errors or 
off nominal conditions. Because each tower’s environment is 
to some extent unique, the specifics of their procedures differ 
from tower to tower. All control techniques are, of course, 
consistent with the regulations cited and described in the FAA 
air traffic control, Order 71 10. 6 5 R, but unique procedures 
and heuristics are passed on to future controllers by onsite 
training. 


The overall tower control process has been formally analyzed 
and modelled including visual and nonvisual components 
(Alexander et al., 1989, Werther, 2006). For example, the 
MANTEA notation (Zografos & Hesselink, 2000) has been 
applied to analyze controller activity in the tower. Some of 
the elements identified in the MANTEA analyses are, in fact, 
visual, but the visual components are only described in very 
general terms such as “visualize runway,” “visualize meteo,” 
etc. These descriptions really only identify the sensory mo- 
dality used to gather the information and a general description 
of the content of the visual information, but they say nothing 
specific about the actual visual viewing conditions or about 
the specific visual stimuli. This feature is, in fact, common in 
other more recent and more sophisticated task analyses of 
visual features seen from the tower. Even the recent model- 
ling done with Petri nets (Werther, 2006) does not identify 
specific visual stimuli but is more connected to estimates of 
time required for the precision with which various visual 
subfunctions maybe executed and to the logical conditions 
and consequences associated with the functions. 

The FAA has done some analysis of the specific visual per- 
formance expected from Tower controllers. The work pri- 
marily focuses on the controller’s surveillance function and 
has been based on visual performance models developed for 
the military by CERDEC at Ft. Belvoir (e.g., Vollmerhausen 
& Jacobs, 2004). These models primarily are intended to 
predict the probability of visual detection, recognition, and 
identification of known targets. “Detection” refers to users’ 
ability to notice the presence of a particular object. “Recog- 
nition” refers to their ability to categorize the object into a 
general class such as a tank, light aircraft, or truck. “Identifi- 
cation” refers to their ability to determine the specific type of 
object, i.e., an Abrams tank, a Cessena 172, or a Ford re- 
fueling tanker. More modem similar visual performance 
models do not require specific calibration techniques to de- 
termine model parameters for specific targets (Watson, Rami- 
rez, & Salud, 2009). 

The CERDEC analysis predicts specific object perception 
from towers of various heights, during a variety of atmos- 
pheric conditions and object distances has been incorporated 
into a web tool to help tower designer ensure that specific 
architectural and sitting decisions for new towers will meet 
FAA requirements. Significantly, this tool also just focuses on 
the surveillance function and does not address the aspects of 
visual motion that tower controllers use for the information, 
separation and safety tasks. 

In order to understand the details of the visual features used in 
tower control it is first necessary to identify the range within 
which controllers use visual information. We can use the 
example of SFO. Informal voluntary discussions with ten 
active controllers who work at this tower were analyzed for 
the physical locations identified as points where various types 
of visual references are taken from approaching or departing 
aircraft (Figure 2.). These points include positions where 
visual contact with the aircraft is first or last considered to 
provide useful information. These positions include those for 
which they come under or leave tower control, where they 


pass important ground references, or where visual contact 
provides other useful information. The points were deter- 
mined independently from each of the controllers in response 
to the question, “When you are in the Local controller posi- 
tion, where are the aircraft when you usefully observe them 
visually, what visual aspects of the aircraft do you observe 
and why?” Controllers could designate more then one point of 
interest for departing and more than one for arriving traffic, 
only two controllers took this option. One point represents 
nine controllers’ overlapping responses identifying approxi- 
mately the same location about 1 n.mi. beyond the end of the 
departure Runway 1 . 

In general it is apparent from the distribution of points that 
controllers’ visual attention is much more spatially distributed 
to the aircraft approaching the 28LR runways and rather ab- 
ruptly drops off about 1 mile off the end of the usual depar- 
ture runways 1LR. These observations refer to the most 
common aircraft flow at SFO but suggest the generalization 
that the local controllers’ visual attention to approaching 
aircraft is distributed over a much large area than that corres- 
ponding to departing aircraft. A likely reason for this is that 
departing traffic is handed off to approach/departure control 
and generally not thereafter of concern to the tower. 

A significant aspect of the controllers’ remarks concerning 
when they first start paying visual attention, or when they last 
pay attention, to aircraft is that they rarely mentioned the 
aircraft’s visual motion 1 . One reason is that for the viewing 
angles and distances to the aircraft approaching SFO, this 
motion is very small in terms of degrees per second, often the 
azimuth rate is on the order of much less than 0.25 °/s and 
rarely more than 0.5°/s. The visual accelerations are even 
much smaller and difficult to see because of atmospheric 
haze, thermal effects, and the visual range beyond 5 miles. 
Visual rates of motion are more important for closer aircraft 
on or near the runways and taxiways. 

Probably the most obvious need for visual contact by con- 
trollers in the tower is to immediately note unusual events that 
are not detected by electronic sensors such as radar. Examples 
could be heavy bird activity or an aircraft leaking fuel onto a 
taxiway. But there are a wide variety of other visual features 
that controllers use on a more regular basis when aircraft are 
close enough for the visual motion to be more easily noticed. 
Discussions with controllers as provided a list of some that 
are used. 

A tabulation (Table 1) of the visual features mentioned in the 
discussions with ten SFO controllers shows the relative 
frequencies with which different features were mentioned. 
These discussions used a “cognitive walk-through technique 
in which the controllers were asked to imagine representative 
approaching, departing, and taxiing aircraft under a variety of 
visual conditions and to report what they looked for visually 
to assist their control tasks. The most frequently mentioned 
features were relative motion between landing or taking off 
aircraft and obstacles that could be on the runway. The first 


1 Visual motion is defined as the angular rate of change of the line of sight 
angle to an aircraft from the tower. 



of these features is probably prominent because SFO has 
intersecting runways commonly used for takeoffs and 
landings. An assessment of all of the features mentioned, 
however, shows what may be a more general element. Seven 
of the 13 features identified in the interviews note that the 
feature helps the controller anticipate future activity. This 
information provides insight into pilot intent, knowledge, and 
likelihood of aberrant behaviour. These predictive cues help 
the controller with the short term planning needed for 
anticipated separation. 

Table 1. Counts of visual features identified in 
interviews with 10 SFO controllers. 


Feature 

# 

Comments 

1 . Relative visual mo- 
tion used to inter- 
leave departures and 
arrivals 

5 

Controllers verify predicted separ- 
ation using relative motion w/r sta- 
tionary references. 

2. Visual check for ob- 
stacles or other a/c to 
verify a clearance 

5 

Obstacle checks include ground ve- 
hicles, aircraft, birds, people 

3. Taxing “With au- 
thority” helps atten- 
tion allocation 

4 

Fast and confident a/c motion al- 
lows controllers to distribute atten- 
tion to pilots who maneuver hesi- 
tantly allowing anticipation of fu- 
ture problems 

4. Aircraft attitude/ alti- 
tude predicts a “Go 
Around” 

4 

Controllers anticipate “Go Around” 
by checking a/c passage through 
various approach gates defined by 
altitude and attitude 

5. Visually apparent ac- 
celeration, speed or 
turn rates 

4 

Controllers mentally integrate mo- 
tion features to anticipate taxiway 
and ground route selection 

6. Visual and radar pos- 
ition and speed are 
cross-checked 

4 

Specific visual land-marks are se- 
lected to cross check radar 

7. Visible wing dip pre- 
dicts coming turn 

3 

Visible banking confirms initial 
conformance to turn clearance 

8. “Mike and a mile” 
rule for interleaved 
take offs and landings 

3 

Predictive rule: Departing A/C 

must be rolling across taxiway Mike 
on RW1 when matched landing A/C 
on RW28 is at least 1 mi out for 
required separation. 

9. Engine smoke or heat 
confirms take off 
start 

3 

Modern engines don’t smoke much 
and have cooler exhaust 

10. Onset of navigation 
lights precedes call to 
tower 

2 

Controllers can anticipate coming 
workload 

11. Visual resolution of 
motion and position 
is better at airport 
than radar 

1 

1-2 nmi. From the tower the “visual 
display” of the real world has more 
“pixels” than associated radar dis- 
p |a >' s 

12. Visual double check 
on a/c tail to verify 
company 

1 


13. Check landing gear 

1 

This is an isolated comment 

probably because it is done 

automatically and is an infrequent is 
issue for major airlines. 


Many of the predictive cues, particularly the motion-based 
cues, are available because of the high dynamic fidelity of 
direct visual contact. Because of noise, sampling artefacts, 
or undue delay these features may not easily be seen on 
electronically mediated information such as that on radar 
displays. A better understanding of exactly what these 
continuous cues are can be developed by examining one 
them quantitatively. An example of such analysis is pre- 
sented below with respect to landing deceleration at SFO. 



SFO_2S110& 


MVIJ864 


resolution: 

1024x 768 

^Markers: 

21.12° 

Marker pixels; 

110,10 849,30 

FOV: 

29.26“ 

Bearing, ■ 

3 true north 

3.25" 

110.1 




Figure 3. The video field of view used for motion tracking 
during the braking phase of landing. 


4. DECELERATION DURING LANDING AT SFO 

The following analysis begins to determine the magnitude 
of this visually sensed deceleration and how it could be used 
by controllers. In doing so we will identify one of the dy- 
namic visual features used in traffic control from the airport 



tower: the change in speed evident during a single glance a 
controller might make towards a decelerating landing air- 
craft 2 . In thinking about what specific aspects of the visual 
stimulus to which the controllers might be attending, it is 
helpful to remember that perceptual discriminations of 
commonly experienced magnitudes of sensory quantities 
such as velocity are fairly well described by Weber’s Law, 
which states that the just noticeable difference is a constant 
proportion of the quantity’s magnitude. This so-called 
Weber fraction generally had a midrange value around 3-5% 
(Goldstein, 2007) for common perceptual discriminations 
such as those that could be made from the control tower for 
landing and departing aircraft. 


velocity profiles (low pass filtered with a 1Hz cut-off) are 
shown in Figures 4. 

Because of the noise present in our current recording tech- 
nique, we were unable to obtain velocity and acceleration 
values with acceptable noise levels. We were, however, able 
to obtain a braking deceleration profile for an A3 19 aircraft 
landing on Runway 28 from the same company, comparably 
loaded and flying in the same wind and weather conditions as 
one of the aircraft we had recorded visually. Since we knew 
the touchdown points for these two A3 19 landings, we’ve 
combined the two trajectories producing what we believe is a 
fairly accurate landing profile as seen from the tower (Fig. 5). 


It is also important to understand that controllers may not be 
directly sensing the visual velocities per se but may develop 
viewing strategies allowing them to translate speed into dis- 
placement during relatively fixed time intervals, thus making 
the detection of unusual rates of change easier. Additionally, 
alterative visual cues to quantities such as deceleration, but 
which are correlated with them, could be used. Aircraft pitch, 
for example, could be a clue to braking. 

It is not so much the visual aspect of the visual information 
that is important as it is the fact that the information revealed 
by vision is relevant, real, direct, unmediated, immediate and 
continuous that makes it possible for the best possible antici- 
pation of future action. This is why the visual input could be 
critical. Replacements for it need to capture the same pre- 
dictive, informational features. 

In order to begin to analyze the visual features actually present 
in real landing trajectories we have initially focused on the 
deceleration profile of aircraft landing on the 28L and 28R 
runways at SFO. Controllers report that they use their sense 
of degree and timing of this deceleration to anticipate which 
taxiway would be needed for the aircraft to exit the active 
runway. This decision is time critical during heavy runway 
use since landing aircraft are staggered in pairs and interleaved 
with departures on crossing runways 1R/1L. 

We have made 15 frame/s video recordings at 1024 X 768 
resolution of the braking phase of 45 aircraft landing on 28L 
and 28R and processed the recordings to measure changes in 
visual velocity. We used a custom MatLab image processing 
technique that isolated the moving contours across a set of two 
frames and averaged them to localize the aircraft and provide 
their screen velocity in degrees per second. Using the viewing 
geometry described in Figure 3, we have recovered the aircraft 
braking profile and computed the changes in visual velocity as 
viewed from the control tower by re-projecting the movement, 
as it would have been seen from the tower. Thirty of these 


2 During normal vision, people make from 3-5 fixations per second 
(Rayner & Castelhano, 2007). However, when studying some 
aspect of an ATC image, fixations duration can increase but rarely 
grow longer than approximately 1.3 s (e.g. Remington, Lee, 
Ravinder, Matessa, 2004). Consequently, a reasonable constraint 
for modeling the duration of a controller’s glance would be to 
insure that they are 1.3 s or less. 



Figure 4. Line of sight (LOS) position and visual velocity of 23 
aircraft landings at SFO showing growing tracking noise after 
~14 seconds. 


Laboratory research has examined the perceptual mecha- 
nisms humans use to perceive 2D acceleration in stimuli 
like our LOS data, and have quantified perceptual thresh- 
olds for acceleration as low as 30 deg/s 2 for integration 
times of 200 ms or more (Calderone and Kaiser, 1989; 
Stone and Ersheid, 2006). From the point of view of the 
control tower, the visual deceleration signals (e.g., the 
deceleration of the visual image of the plane) are much 
lower (e.g., <1 deg/s 2 ) than these threshold values, and 
highlight the clear distinction that must be made between 
deceleration of the visual image of the aircraft (e.g., the 
slope of visual velocity traces in Figs. 4 and 5) and the 
higher-order inference that the ground speed of the plane 
on the runway has decreased. Although these data allow 
us to rule out the idea that the former supports the latter, 
the higher-order mechanisms of inference supporting per- 
cepts of deceleration in the 3D world likely “re-project" 
velocity signals on the retina onto a mental model of the 
runway, allowing the construction of a deceleration per- 
cept. The present data do not speak to how this might 
occur, although design concepts for remote towers will 
have to identify the types of visual information that will 
allow controllers a robust percept of motion within the 3D 
world model. 
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Figure 5. Line of sight changes for a single medium jet transport 
landing at SFO. 


The deceleration profile in Figure 5 shows the aircraft 
approaching and passing the tower as it decelerates. In 
fact, during the approach the visual velocity actually in- 
creases during the deceleration because of the decreas- 
ing distance between the aircraft and the tower. It is 
clear from the deceleration profile that there are several 
phases of braking due to deployment of the thrust re- 
versers, spoilers, and mechanical brakes and further data 
collection and processing needs to be done to more pre- 
cisely identify these periods. However, the very smooth 
velocity plot in Figure 5 (third panel from top] already 
shows that the amounts of velocity change in the braking 
within any short time window 2 s or less are well less 
than the 3-5% Weber fraction for a just noticeable dif- 
ference. This level is defined by convention to be that 
difference in a sensory quantity that can be detected cor- 
rectly 75% of the time and is therefore not evidence of a 
very strong sensory stimulus. 


This observation leads to some skepticism that the con- 
trollers are detecting velocity change per se, but rather 
may have developed a strategy to detect speed change by 
some other means, perhaps by comparing displacement 
for approximately equal time periods. Such a timing 
strategy might be evident in eye tracking records of con- 
trollers judging aircraft deceleration. Of particular inter- 
est will be future analyses and experimental to deter- 
mine how well the controller's sense of aircraft decelera- 
tion can be maintained with airport imagery spatially 
degraded by pixelation and sensor noise, and temporally 
degraded by low sampling rate. These issues will be ad- 
dressed by future research. 
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